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(57) ABSTRACT 

A signal is encoded, for example, perceptually and, during 
or after the perceptual coding process, a digital watermark is 
inserted into a quantized digital information signal resulting 
from the perceptual coding process in such a manner that its 
insertion is imperceptible to one later listening to, displaying 
or otherwise utilizing the information signal. Moreover, the 
digital watermark may be inserted in accordance with a key 
indicative of the location of the mark in the digitally encoded 
signal. The key may be protected with a trusted entity and 
distributed in such a manner as to be not detectable by a 
pirate. Consequently, the key may be utilized at watermark- 
ing apparatus that can be located anywhere in the distribu- 
tion channel of a copyright protected work. The key may be 
embedded in a secure microprocessor of validating appara- 
tus at a point of distribution or even at a point of sale. In this 
manner, when or after a digital bitstream is fixed in a 
medium of expression or broadcast, the watermark can be 
applied at the time of fixing in a copy, thus validating an 
original legitimate copy. For example, a compact disc 
recording may be validated at the point of sale as an 
authorized copy through watermarking (validating) appara- 
tus according to the present invention or, at the time of 
broadcast over radio frequencies or distribution via the 
Internet, information may be watermarked. In this manner, 
even though a work may be pirated, further information may 
be included or otherwise indicated by the watermark to 
assist in, not only recognizing a pirated work and the 
original from which the pirate copy was produced, but the 
identity of the pirate who later makes an unauthorized copy 
of the original. 

36 Claims, 4 Drawing Sheets 
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METHOD AND APPARATUS FOR 
WATERMARKING DIGITAL BITSTREAMS 

This application claims priority to and is based in part on 
provisional application No. 60/067,225 filed Dec. 3, 1997 
and entitled "On Combining Watermarking with Perceptual 
Coding" of J. Lacy et al 

BACKGROUND OF THE INVENTION 

1. Technical Field 

The present invention relates to the field of protecting 
copyrighted works from piracy by fingerprinting or water- 
marking the copyrighted information and, more particularly, 
to a method and apparatus for inserting hidden data such as 
a digital watermark into a signal representing a copyrighted 
work simultaneously with or immediately after 1) percep- 
tually encoding a digital representation of an audio signal or 
2) perceptually or otherwise coding a digital representation 
of a video signal or other information carrying signal. 

2. Description of the Related Arts 

Piracy of copyrighted works is a serious problem. By way 
of example, street vendors in metropolitan areas offer 
videotapes, audio compact discs, audio tapes and the like 
which appear on their surface to have been properly distrib- 
uted and offered for sale with the permission of or under 
license to the copyright owner, when, in fact, they are not. 
Computer software and games are distributed freely via the 
Internet and pirate copies abound. There are many ways a 
pirate can obtain free access to a copyrighted work that is 
original. By "original" as intended herein, we define "origi- 
nal" as an original or master copy from which other copy- 
right owner authorized copies have been made or an "origi- 
nal" is an authorized copy made from the original or master 
copy. All other copies will be defined herein as "pirate" 
copies and comprise copies that are not authorized by a 
copyright owner. 

By way of example of the ease with which a pirate is able 
to make a pirate copy, the pirate can produce a high quality, 
perhaps, perfect reproduction from an original that they have 
legitimately purchased or otherwise legitimately obtained, 
can record a live performance for which copyright protec- 
tion is asserted or can record from a broadcast performance 
of a live or copied performance. Other means of pirating are 
best known to the pirates themselves. 

It is an objective of the present invention to protect a 
copyrighted work. Protection methods are known. Despite 
efforts by pirates to make a copyrighted work appear to be 
an original, a method is known to attempt to hide in the 
copyrighted work itself indicia of the copyright owner's 
identity and source. The burying of such indicia perhaps 
began with early map-makers who would "fingerprint*' their 
work by intentionally placing an error in the map. The map 
maker could quickly look at a copy, recognize their finger- 
print (intentional error) and then accuse the individual 
offering the map of infringement, knowing the offeror to not 
be authorized to do so. Today, it is known to embed in 
integrated circuits a fingerprint or indicia of ownership that 
may appear on the surface to comprise a circuit component, 
when in fact, the indicia comprises a fingerprint intended by 
the owner to deceive a pirate. 

With works created on paper, it is known to "watermark" 
the paper itself with indicia of its source. For example, since 
the mid-1 800's, it became known and desirable to protect 
the authenticity of such items of value as postage stamps and 
currency from copying/counterfeiting by watermarking the 
item of value, A watermarked item of value or legal docu- 
ment can both identify and verify the source of the docu- 
ment. 
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As used herein, by fingerprinting and watermarking, no 
distinction is intended between them regardless of their 
historical distinctions. Watermarking as used herein is 
intended to refer to either term and to signify the same 
5 process; namely, it is an objective of watermarking to 
identify and to verify the source of an original as defined 
above. 

According to the prior art, it is not only known to use 
watermarking for works created on paper but also for 

10 protecting analog or digital signals representative of a copy- 
righted work. For example, in binary digital bitstreams for 
transmitting digital audio signals in a cable television 
system, it is known to identify, for example, the title, track 
and artist in a header portion of blocks of data representative 

1S of a copyrighted work. The problem with such systems is 
that the location of the header carrying such a watermark can 
be easily become known to a would-be pirate. It always 
appears at the beginning of the work. Consequently, the 
watermark can be easily removed. 

20 Digimark Corporation of Portland, Oregon offers a 
method of watermarking images that may represent copy- 
righted works. In particular, pixels within digital photo- 
graphic images carry a digital watermark prior to distribu- 
tion of the images, for example, over the Internet. By hiding 

25 the watermark within the image itself, it is expected that a 
pirate will have difficulty locating the watermark. 
Consequently, it is a further objective of the present inven- 
tion to not only apply a watermark but apply a watermark in 
such a manner as to disguise its location from a pirate so the 

30 pirate cannot simply remove it. 

In the art of coding copyrighted works, there has been a 
recent movement toward perceptual coding. Perceptual cod- 
ing relates to coding and decoding designed to recognize 
that the signal fidelity perceived by humans is a better 

35 quality measure than "fidelity" computed by traditional 
distortion measures. Traditionally, for example, it might be 
a measure of quality to assert that a speaker has a flat 
frequency response across the spectrum of frequencies 
between 0 frequency and 20,000 Hertz. On the other hand, 

40 the human ear may not be able to even hear frequencies at 
the lower or higher end of the spectrum. Now, for example, 
in audio, quality is defined to mean "listener indistinguish- 
able from compact disc audio," generally regarded today as 
the best quality audio in comparison with tape, record or 

45 earlier forms of recording a live performance. 

Patents describing audio perceptual coding include U.S. 
Pat. No. 5,285,498 of Johnston issued in 1994. FIGS. 1 and 
2 of the '498 patent generally describe prior art perceptual 
coders, typically involving the conversion of analog data to 

50 digital, a transformation of the digital data in association 
with a perceptual model and a quantizer. Recent articles on 
perceptual encoding/decoding include: "AT&T Perceptual 
Audio Coding (PAC)" of Johnston et al., Collected Papers 
on Digital Audio Bit-Rate Reduction, Feb. 13, 1996 and the 

55 following three papers "MPEG-2 Audio Stereo and Multi- 
channel Coding Methods" by Johnston et al.; "ISO/IEC 
MPEG-@ Advanced Audio Coding" by Bosi et al.; and 
"Enhancing the Performance of Perceptual Audio Coders by 
Using Temporal Noise Shaping (TNS)" by Herre et al., all 

60 published with Conference Papers of the 101" Convention 
of the Audio Engineering Society, November 8-11, 1996. 
The recent papers describe alternative embodiments and 
enhancements to the original perceptual coding apparatus 
and methods described, for example, by Johnston in the '498 

65 patent and his predecessors. Recently filed patent applica- 
tions on related subject matter incorporated by reference 
herein include U.S. Pat. No. 6,111,844, issued Aug. 29, 2000 



11/10/2003, EAST Version: 1.4.1 



US 6,330 ; 

3 

entitled "Quality Degradation Through Compression/ 
Decompression" and U.S. application Ser. No. 08/888,014 
entitled "Custom Character Coding Compression for Encod- 
ing and Watermarking Media Content," mailed Jul. 3, 1997. 
Also, this application claims priority to U.S. provisional 5 
application Ser. No. 60/067,225 filed Dec. 3, 1997 entitled 
"On Combining Watermarking with Perceptual Coding/' 
This application, converted to a regular application, has been 
published as W099/29114 on Jun. 10, 1999. 

By perceptual coder as used herein is intended a general 10 
form of coder which may comprise simple perceptual coders 
or more complex perceptual coders, for example, hybrid 
coders, integrated source-model perceptual coders and per- 
ceptual coders using temporal noise shaping among other 
coders. 15 

Information carrying signals that have high information 
content do not appear to be susceptible to elimination of 
"irrelevance" and perceptual coding. For example, a digital 
data computer program may have an undiscovered and so 
not particularly "relevant" embedded subroutine. 20 
Nevertheless, compression principles may still be applied to 
such a signal to eliminate redundancy. Yet, perceptual cod- 
ing may be applied to such a signal for at least the latter 
reason. 

25 

The ability to easily replicate digital audio, video, com- 
puter program, game and other data, protectable by 
copyright, by simply copying binary bitstreams is a major 
impediment to the use of the Internet or other telecommu- 
nications channels for content distribution/sale. 3Q 
Consequently, it is a further objective of the present inven- 
tion to eliminate such an impediment to commerce over the 
"information superhighways" of the future. 

Thus, in view of the above, it is an object of the present 
invention to provide apparatus and a method of watermark- 35 
ing a video television signal as well as an audio or other 
information carrying signal and open up the information 
superhighway to free commerce in protected data. 

SUMMARY OF THE INVENTION 

40 

According to the principles of the present invention, a 
signal is encoded, for example, perceptually and, during or 
after the perceptual coding process, a digital data sequence, 
for example, a digital watermark is inserted into a quantized 
digital signal resulting from the perceptual coding process in 45 
such a manner that its insertion is imperceptible to one later 
listening to, displaying or otherwise utilizing the signal. 
Moreover, the digital watermark may be inserted in accor- 
dance with a key indicative of the location of the mark in the 
digitally encoded signal. The key may be protected and 50 
secured with a trusted entity and distributed in such a 
manner as to be not detectable by a pirate. Consequently, it 
is a principle of the present invention that the key may be 
utilized at watermarking apparatus that can be located 
anywhere in the distribution channel of a copyright pro- 55 
tected work. The key may be embedded in a secure micro- 
processor of validating apparatus at a point of distribution or 
even a point of sale. In this manner, when or after a digital 
bitstream is fixed in a medium of expression or broadcast, 
the watermark can be applied at the time of fixing in a copy, 60 
thus validating a copy. For example, a compact disc record- 
ing may be validated at the point of sale as an authorized 
copy through watermarking (validating) apparatus accord- 
ing to the present invention or, at the time of broadcast over 
radio frequencies or distribution via the Internet, informa- 65 
tion may be watermarked. In this manner, even though a 
work may be pirated, further information may be included or 
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otherwise indicated by the watermark to assist in not only 
recognizing a pirated work and the original from which the 
pirate copy was produced but the identity of the pirate. 

In U.S. Patent Application Serial No., entitled "Custom 
Character-Coding Compression for Encoding and Water- 
marking Media Content" (Attorney Docket Number Lacy 
3-5-6) filed Jul. 3, 1997, it has already been suggested to use 
designated symbols in one or more Huffinan codebooks for 
embedding a watermark in a compressed bit stream. The 
value of the watermark bits recovered from the bitstream 
depend either on the values associated with the symbols or 
the position of the symbol in the compressed bit stream. 

Thus, the present invention also includes dependent fea- 
tures directed to the information contained within the water- 
mark itself. Not only will the watermark identify the work, 
identify the source of the work, and identify the author but 
may identify variable information such as the date and time 
the original was distributed or broadcast, the identity of the 
seller, the identity of the purchaser and the location from 
which the original was distributed or broadcast. While 
purchase/sale is presumed in this comment, the actual form 
of contractual arrangement may differ between parties in the 
distribution channel. For example, the work may be legiti- 
mately sold, licensed, leased for a period of time and the like 
all within the boundaries of being an original. Consequently, 
an applied digital watermark may further comprise indicia of 
the form of the transaction, and, for example, if the trans- 
action represents a lease, the expiration date of the lease. If 
the transaction is a limited play license defined by a number 
of plays, the watermark may comprise data indicating the 
number of licensed plays. On the other hand, from the 
transaction date or expiration date forward or after the 
number of licensed plays has been spent, if a copy of the 
original appears after that date, the copy can be identified as 
a pirate copy and the pirate identified. 

The invention also considers the player, computer or other 
device upon which a legitimate recipient or pirate uses or 
plays the protected work. The device may require the key to 
play the work. The key may be extracted from the com- 
pressed digital bitstream or distributed personally, for 
example, as a memory or "smart" card or the decoder require 
both kinds of keys to operate. Even if the key is provided, 
the device may comprise a time-of-day clock that, upon 
expiration of the period of the lease, actuates the device to 
preclude further play. 

In summary then, apparatus for inserting a digital water- 
mark into protected information comprises a perceptual 
coder for coding an information signal representative of the 
protected information, a watermark location selector respon- 
sive to the perceptual coder for selecting a location for 
inserting a digital watermark, a digital watermark generator 
for generating a digital watermark and a quantizer of the 
perceptual coder responsive to the digital watermark detec- 
tor and the watermark location selector for inserting a digital 
watermark at selected locations within an output bitstream 
including the perceptually coded information signal. 

A method for inserting a digital watermark into protected 
information comprises the steps of perceptually encoding an 
information signal representative of the protected 
information, selecting, responsive to the perceptual coding 
step, a location for inserting a digital watermark, generating 
a digital watermark, and during perceptual coding, quantiz- 
ing the information signal to insert the generated digital 
watermark at selected locations within an output bitstream 
including the perceptually coded information signal. 

Other features and advantages of the present invention 
will become clear from an understanding of the following 
detailed description with reference to the drawings. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a block schematic diagram of apparatus for 
watermarking copyright protected information according to 
the present invention comprising a perceptual encoder, a 
watermark location (index) selector responsive to transfor- 5 
mation processes of the perceptual coder for selecting a 
location within the perceptual coded bitstream for a water- 
mark and a digital watermark generator, a quantizer of the 
perceptual coder outputting a digitally watermarked bit- 
stream with a watermark inserted at the selected location; io 
FIG, (a) provides details of a generalized digital encoder and 
FIG. 1(b) provides more specific details directed to the 
design of a particular digital encoder involving the encoding 
of an analog signal. 

FIG. 2 comprises a typical power versus frequency spec- 15 
tram of a trumpet useful for explaining principles of per- 
ceptual coding. 

FIG. 3 is a functional block schematic diagram of a 
decoder/player for use in playing a digital bitstream of 
information having an embedded, hidden data watermark. 

DETAILED DESCRIPTION 

Now the present invention will be described referring to 
the drawings for one exemplary embodiment of the inven- 
tion. FIG. 1(a) represents a block schematic diagram of a 
generalized digital encoder for embedding a hidden data 25 
stream into a digital information signal to be perceptually 
encoded according to the present invention. FIG. 1(b) rep- 
resents a specific analog signal implementation of FIG. 1(a) 
including specific recommendations for implementation of 
encryption. Similar reference characters used in each draw- 30 
ing represent similar components. As has been described 
above, perceptual coding has evolved over the past several 
years from a concept to practical application. Moreover, 
several enhancements to perceptual coding/decoding tech- 
niques have evolved which describe enhanced or improved 35 
performance in one or another step or component of the 
encoding/decoding process. 

Referring to FIG. 1(a), there is shown a perceptual coder 
ilO, a hidden data index selector 120 for selecting an index 
or location to insert a hidden data sequence, for example, a 40 
digital watermark, into an information signal that is being 
perceptually coded and a digital hidden data generator 130. 
The entirety of perceptual coder 110 is drawn to generically 
describe the process of perceptual coding and is not intended 
to describe the most current or best manner of implemen- 45 
tation. The method and means for perceptual encoding does 
not comprise the present invention, rather it is the manner 
and apparatus for inserting (and when appropriate, 
removing) a hidden data sequence such as a digital water- 
mark into the coded, compressed signal as will be further 50 
described below. Each of the components of the perceptual 
coder 110 is known from the prior art and may comprise 
hardware, a software programed processor or firmware io 
implementation. Similarly, each of the components of the 
hidden data index selector 120 and digital hidden data 55 
generator 130 is known from the prior art individually. It is 
the combination of elements described by the claims which 
comprises the present invention. 

An uncompressed digital data stream is presented to 
perceptual coder 110 at point 100. The uncompressed digital 60 
data stream may comprise audio, video, software algorithm, 
or other protectable digital data stream. What is intended by 
"protectable" is the protection of intellectual property by 
copyright, for example, any work that has been reduced to 
a tangible medium of expression including, but not limited 65 
to, audio, video, software or other work protectable by 
copyright. 
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The first task of the perceptual decoder as represented by 
step 80 is the generation of N sample points for the input 
digital stream of protectable information according to a 
compression algorithm. The compression algorithm is typi- 
cally predetermined or selectable according to the type of 
presented data. For example, audio may be compressed 
differently from video data and the like. As a result of the 
sample point generation step 80, N samples are presented to 
an N-point transform step 104. N-point transform step 104 
may comprise any one of a plurality of possible N-point 
transformations as will be further described herein. The 
result of the N-point transform step is N coefficients which 
are provided to quantization step 140 and to perceptual 
threshold determination step 105. Perceptual threshold 
determination step 105 may vary among types of informa- 
tion signals such as audio and video. Nevertheless, the 
objective of the perceptual threshold determination is a 
threshold for human perception such that below the thresh- 
old may be data points that are not perceptible to human 
perception but above the threshold are data points that are 
perceptible to human perception and thus represent data 
points that during compression and the quantization process 
may not be discounted. 

Quantization step 140 may be generally described as the 
process of manipulating perceptual coding coefficients to 
insert a digital water mark into the perceptually coded 
bitstream. Quantization step 140 in one embodiment of the 
present invention represents three steps. The first and second 
of these relate to the expected tasks of the perceptual coder. 
The first step is to select a scale factor. The quantization 
scale factor may be different for different types of data, such 
as audio or video. The scale factor selection may be respon- 
sive to the perceptual threshold determination process. Once 
the scale factor is selected for the input N coefficients, the 
coefficients are quantized according to the selected scale 
factor. The output is N quantized coefficients representing 
the compressed digital bitstream. 

The non-typical third step of quantizer 140 is the process 
of injecting a hidden data sequence or stream in indexed 
locations of the output compressed data stream according to, 
for example, an N-Point transformation process 104 as will 
be further described herein. The hidden data stream may 
represent a watermark for not only marking protected infor- 
mation as protected but assisting one in tracing a pirate of 
the protected information as will be further discussed herein. 
Hidden data index selector 120 responds to the N-point 
transform process 104 by identifying locations in the com- 
pressed digital information stream that a watermark may 
occupy. Digital hidden data generator 130 provides the data 
to be introduced or inserted onto the compressed informa- 
tion stream at the selected indices. 

The output of quantizer 140 is N quantized coefficients 
representing a compressed information data stream includ- 
ing digital hidden data. The data stream is presented to an 
entropy encoder 107. Entropy encoder 107 performs the 
final stage of compression in a lossless (invertible) manner 
by encoding frequently occurring symbols using shorter 
codewords and less frequently occurring symbols using 
longer codewords. The use of shorter codewords exceeds the 
use of longer codewords and hence there is compression. 

Entropy coder 107 provides an output which fills an 
output buffer 108 of compressed digital information signal 
(including hidden data). If the buffer is full, a buffer fullness 
indicator is fed back to the quantizer 140 to throttle or stop 
the quantization process. Otherwise, the output buffer 108 
outputs an output compressed bitstream for, for example, 
recording, broadcast or other signal distribution. 
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One skilled in the art will be familiar with a variety of 
embodiments for implementing quantizer block 140 includ- 
ing all three steps of selecting a scale factor, quantizing 
coefficients and injecting the hidden data (watermark). Two 
embodiments are known for audio (the Advanced Audio s 
Coding, AAC and Perceptual Audio Coding, PAC audio 
compression algorithms). Also, further embodiments are 
described for known (EG -2) video compression algorithms 
in the above identified U.S. provisional application ser. No. 
60/067,225 filed Dec. 3, 1997. 10 

In both AAC and PAC audio compression algorithms, 
coefficients (also known as spectral values) are grouped into 
scale factor bands that are quantized with the same scale 
factor. Once coefficients are grouped into scale factor bands, 
an algorithm is used to quantize them according to the 15 
present invention. Distinct hidden data (watermark) bits can 
be inserted into each coefficient in a scale factor band that 
has at least one non-zero coefficient. As can be seen, the 
objectives, in either an audio or video embodiment, are 
achieved of embedding hidden data according to the modi- 20 
fied scale factors generated by the algorithm into the output 
compressed information bit stream. 

Now referring to FIG. l(b\ FIG. 1(b) represents an 
embodiment for perceptually encoding an analog signal 
containing protectable information and provides particular 25 
recommendations for generating a hidden data index and 
encrypting the hidden data (watermark). Generally, an ana- 
log signal for encoding is received into perceptual coder 110 
at analog source 101. The analog signal comprises according 
to the present invention a signal representing protected 30 
information. The protected information itself may represent 
an audio performance, a video or visual performance, a 
computer software algorithm, a bitstream representing a 
book or novel or even be descriptive of other sensory signal 
(taste, touch or smell) that may be capable of perceptual 35 
coding. By way of example, the analog source may be a 
source of a combination of audio and visual signals and so 
represent the protected information content of a movie, 
computer game or live audio/visual performance such as a 
ballet or opera. 40 

typically, a low pass or other filter 102 is applied for 
filtering noise or segregating the bandwidth of the analog 
signal or related purpose. In this manner, the analog signal 
presented to the analog to digital converter 102 can be 45 
considered to be relatively pure of adverse influences and 
confined to a known or expected bandwidth of the signal. 

Analog to digital converter 103 converts the analog signal 
(for example, voice or visual) to digital data. The digital data 
output from the AID converter 103 may be considered a 50 
faithful representation of the filtered analog signal, except in 
binary form. Of course, if the data presented to the percep- 
tual coder is already in digital form, neither steps 102 or 103 
are required, for example, if the input is a digital computer 
software algorithm or game program. 55 

Block 104 represents the transformation block of a per- 
ceptual coder, which for example, according to the prior art, 
may comprise an N point fast Fourier transform or a 
Modified Discrete Cosine Transform (MDCT) or other 
transform for perceptual coding. These are variously imple- 60 
mented according to the prior art; for example, an MDCT 
transform is implemented via a filter bank for transforming 
audio signals according to the '948 patent. The encoding 
process slices the time series into contiguous time segments 
and transforms each segment into the frequency domain. 65 
Once in the frequency domain, the spectral coefficients are 
compared with a perceptual threshold function T(f). This 
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comparison step is shown in the diagram as calculation of 
perceptual thresholds step 105. If the coefficients are below 
the thresholds, the coefficients potentially can be discarded 
without any loss in perceived quality. Yet, these coefficients 
represent candidate locations for placement of at least a 
portion of a digital watermark. If above the thresholds, the 
coefficients are quantized and output as necessary data 
representative of the original analog signal 101. 

Referring briefly to FIG. 2, for example, there is shown a 
power versus frequency spectrum for the sound of a trumpet. 
Also, shown is the perceptual threshold function T(f) shown 
as a stair step curve C over the sound wave W. A great deal 
of the trumpet signal is below the perceptual threshold and 
may be discarded in a perceptual coder. All of these signal 
portions below the threshold function are candidate water- 
mark locations. For example, humans can hardly, if at all, 
hear an audio signal at very low frequency. Neither can we 
humans hear an ultrasonic frequency. The peak in the 
threshold function T(f) model shown rises rapidly and then 
decreases steadily from approximately 1000 Hz out to 
12,000 Hz. Thus, the highest fidelity sound may be consid- 
ered wasted on the human ear and perceptibly equivalent 
quality can be achieved by only sending data indicative of 
the signal above the threshold function. 

The quantizing step 106 follows the perceptual coding 
transformation. What is above the threshold is quantized and 
output as necessary data for reconstruction of the original 
analog signal 101. Quantization is the three step process 
described above in considerable detail. The output of the 
transformation process 104 is operated upon by the coeffi- 
cient quantizer 106 in accordance with the calculated per- 
ceptual thresholds of the threshold function output from 
calculation block 105. For example, in an audio signal such 
as a power level of 40 dB at 1,000 Hz, it may be appropriate 
to quantize that level using a high resolution quantizer 
having a larger number of steps, for example, 8 or 16 bit 
coding. In perceptual coding, one probably would not quan- 
tize a DC signal at all, because the human ear cannot 
perceive 0 frequency, regardless of the power level. 

The binary output of the coefficient quantizer 107 is, in the 
depicted embodiment, provided to an entropy encoder 107, 
already described above. The output of the entropy encoder 
is the digital bitstream that may be recorded on disc, tape or 
otherwise recorded or may be broadcast, transmitted or 
otherwise distributed from the point of sale or distribution. 
The output buffer 108, described above, is believed unnec- 
essary especially in an embodiment contemplating immedi- 
ate distribution at higher data rates than the rate of the 
compressed data stream output of entropy encoder 107. 

In accordance with the present invention, the output 
binary bitstream contains a recoverable watermark gener- 
ated by generator 130 and located in the bitstream according 
to the output of watermark location (index) selector 120. It 
is assumed that it is possible and desirable to alter some bits 
of an encoded audio, video or other digital bitstream without 
affecting the perceived quality of what is played out. A 
digital signature is created for a specific protected media 
stream. The signature may be a function of a number of 
identifying parameters as will be further described below, 
for example, associated with a specific location in the 
distribution channel, date and time of distribution and type 
of distribution transaction. The digital watermark or signa- 
ture is created by encrypting the chosen set of parameters. 
The encryption key is intended to be kept secret and pro- 
tected and secured by a trusted entity and only distributed in 
a protected manner, for example, embedded in a secure 
manner (for example, in a secure microprocessor) or dis- 
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tributed encrypted itself with yet another key. The key, as 
will be described below, may be provided to a legitimate 
recipient of an original (for example, if authentication must 
take place before the data can be decoded.) The watermark 
location (index) selector 120 operates via a signal from the 5 
transformation function 104 as to what coefficients are 
below the threshold. The above process of transformation 
104 leads to a sequence of spectral samples whose coeffi- 
cients are below the perceptual threshold which may 
referred to as a candidate watermarking sequence B(i). The 10 
sequence generated at block 124 represents all candidate 
opportunities for inserting watermarking data. The sequence 
is fed to a logical AND block 125 which receives an input 
that is generated, in one embodiment of the present 
invention, involving a seed 121 for generating an encryption 15 
key K and a selector 122 for selecting a desired percent of 
watermarking overhead. By watermarking overhead is 
intended the degree of watermarking, for example, measured 
by the amount of watermark information to be carried by the 
watermark as well as other factors. 20 

A C(i) binary output sequence of a random number 
generator (RNG) 123 may be generated by any one of a 
number of approaches. One known approach involves con- 
trolling the generator by means of an input percent or 
probability. In accordance with the present invention, we 25 
assume the addition of some percent of watermarking 
overhead, for example, the probability of C(i)=l equals 
x/100, where xAOO, for example, may be 1%. Also assume 
a random number generator 123 generates numbers uni- 
formly between 0 and m. For each member of the candidate 30 
watermarking sequence output at candidate generator 124, 
the random number generator 123 is called to output a 
random number. If the output is greater that (l-x/100)*m, 
then, set C(i)=l; otherwise, we set C(i)=0. Other ways of 
generating a C(i) sequence may come to mind of those 35 
ordinarily skilled in the art as well. 

There are likewise various possibilities for encoding the 
digital watermark data via watermark generator 130 as well. 
For example, where C(i), as above, equals one, and if T(f), 
the threshold function, is the perceptual threshold at spectral 40 
coefficient f for the ith member of the candidate watermark- 
ing sequence, then a one may be encoded as 2T(f)/3 and a 
zero as T(f)/3. 

There are likewise various possibilities for encoding the 45 
digital watermark data via watermark generator 130 as well. 
For example, where B(i), as above, equals one. For example, 
if T(f), the threshold function, is the perceptual threshold at 
spectral coefficient f for the ith member of the candidate 
watermarking sequence, then a one may be encoded as 
2T(f)/3 and a zero as T(f)/3. 

Referring again to watermark selector 120, the seed and 
the watermark overhead selection are used as inputs to a 
random number generator 123 which outputs a binary 
sequence. While a random number generator (RNG) 123 is 55 
depicted, the generator 123 may comprise a pseudorandom 
number generator or other more predictable generator than 
one that is assumed to be more random, especially when it 
operates responsive to a seed. The output binary sequence of 
the RNG 123 and the candidate sequence B(i) are ANDed at 60 
logical AND gate 125 and its output result is used to index 
where in the quantizer 106 output a digital watermark is to 
be inserted. 

Now an exemplary hidden data (watermark) generator 

130 will be described with reference to FIG. 1(b). The digital 65 
hidden data (watermark) generator 130 comprises an input 

131 of a plurality of watermark parameters. For example, 
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watermark input parametric data may comprise relatively 
constant data such as data indicating the owner of the 
protected content and the identity of the protected work and 
relatively variable data, for example, the distributor, the 
customer name, the date and time of transaction, the date and 
time of making of a master, the type of transaction (gift, sale, 
license or lease), date of expiration, number of permitted 
plays or other parameter relating to the expiration of any 
license or lease, the encoding algorithm that was used to 
encode the signal via perceptual coder 110 or any other 
related parameter. No particular order is asserted for the 
parametric data of the watermark. Nevertheless, data repre- 
senting the various parameters should be advantageously 
grouped upon input. For example, it may be appropriate to 
collect variable data such as point of distribution data 
together (time, date, type of arrangement, identity of 
recipient, location of distribution) in a separate location 
from constant data such as copyright owner and identity of 
the work, for example, by title, track and artist. Some of the 
data, such as customer identity and date and place of 
transaction, serves the purpose of assisting one in tracking a 
would-be pirate. Other data serves to identify the protected 
information. Yet other data, depending, for example, on the 
transaction, may prevent a player from further playing or 
otherwise decoding and utilizing the compressed data stream 
of protected information. 

In the depicted embodiment of FIG. 1(b), the parametric 
data of watermarking parameters entered at block 131 is 
encrypted with key K at block 132 which may be the same 
key K generated by seed 121 or a different key. Of course, 
the use of such an encryption key K protects the parametric 
data from piracy or tampering. The encryption may be by 
any known means and results in a multi-bit digital water- 
mark at block 133. Encryption processes that may be utilized 
include well known processes such as the DES algorithm, 
the Cylink algorithm, the algorithm known as pgp (pretty 
good privacy) and other algorithms known in the art. The 
generated watermark is input to coefficient quantizer 106 
and inserted at the indexed locations selected by the water- 
mark index selector 120. Also, in alternative embodiments, 
the encryption may involve two, three or more tiers of 
encryption, for example, involving keys which change peri- 
odically as well as fixed keys determined at the time of 
manufacture (such as secret serial number of a decoder) and 
are distributed to points in the distribution channel in a 
protected manner, such as keys of the month. In alternative 
embodiments, memory cards or smart cards including a 
secure microprocessor with embedded decryption algorithm 
may be utilized to distribute new keys periodically or 
different encryption algorithms for use in decoder devices 
distributed in the distribution channel to keep pirates guess- 
ing. 

It may now be apparent that the watermark is inserted in 
such a manner that it is imperceptible and has no impact on 
the perceived quality of a coded analog signal 101. 
Moreover, the watermark itself is not recognizable, even as 
a digital binary bitstream because 1) it has been encrypted 
with a key K and 2) the watermark is distributed throughout 
the output bitstream of quantizer 106 according to a selected 
percent of watermarking overhead and a seed. It is not easily 
recognizable and locatable, for example, at the beginning or 
end of the protected work. Of course, the seed, the key, the 
selected watermarking overhead, even the characteristics 
and order of the parametric data comprising the watermark 
are unknown to the pirate and, consequently, must be 
generated by the pirate in order to be able to unwatermark 
the signal. If the pirate is the purchaser and freely distributes 
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copies without authorization, unauthorized copies may decoder 310 outputs a decoded bitstream which is presented 

include the purchaser's identity in the watermark. to watermark detector 315 and for presentation, for example, 

Also, it is important to note that in furtherance of the f° r listening, for playing or for display or other output at 

objective of the present invention that the watermarking of presentation box 320. The watermark detector 315 first 

a protected signal may occur anywhere in the distribution 5 detects the location of the watermark and then extracts the 

channel. At such a point of distribution, all components of watermark at the indexed locations. As already indicated, the 

the perceptual coder are not required for watermarking a watermark may indicate an expiration date or a finite num- 

signal. The digital data may already be coded and recorded. ber of plays. If the expiration date has expired or the number 

It is then fed through a generator for generating a candidate of plays exceeded, then an authorization signal to deny play 

watermarking sequence 124 and requantized and restored 10 is presented to presentation step 320; otherwise, play is 

with a digital watermark or watermark portion, permitted by the authorization signal. 

It may be desirable to extract the watermark from the there has beeD shown and described a method and 

encoded data stream. For example, an already included apparatus for digitally watermarking a binary bitstream 

watermark may be extracted and new data added to the output of a perceptual coder, simultaneously with or after 

digital watermark, for example, at a point of distribution. As 15 perceptual coding that meets the objectives sought. Any 

another example, a personal computer disc player or other United States Patents or patent applications referenced 

means of utilizing the digital bitstream may extract the herein should be deemed to be incorporated by reference as 

watermark and be operative to deny use of the recorded data, lo mc i r entire contents. Variations and alternative embodi- 

In such situations, what will be necessary, depending on the ments of the present invention which may become readily 

level of encryption applied, is to have access to the key K or 20 apparent to one of ordinary skill reading the above descrip- 

keys, the random number sequence and the perceptual tion of tne invention should be deemed to be encompassed 

coding function applied at transformation and perceptual within its scope which should only be construed to be 

threshold calculation blocks 104 and 105 respectively. limited by the claims which follow. 

In an alternative embodiment of a watermark extractor, ]^ hat we clau ? K ; 

the watermark may be extracted from the decoded bitstream 25 1 Apparatus for inserting a digital watermark into pro- 

by transforming the decoded time series into the frequency tected ^formation composes 

domain, extracting the watermark bits using the watermark- a transformation function of a perceptual coder for coding 

ing sequence B(i) and decrypting using key K. Multiple an information signal representative of the protected 

frequency domain transformations may be carried out based 3Q information, 

on various encoding parameters (segment length, encoding a watermark location selector responsive to the perceptual 

rate and the like) and watermark extraction steps before the coder for selecting a location for inserting a digital 

correct combination of encoding parameters corresponding watermark, said watermark location selector including 

to the original watermarking are arrived upon. a logical gate responsive to a random number 

In a further alternative embodiment, it may be advanta- 35 generator, said random number generator, responsive to 

geous to detect and extract a watermark in the decoded a seed and a selection of a watermark overhead, for 

bitstream to intentionally insert a synchronization marker at outputting a binary sequence, 

the beginning of the stream. (Placing synchronization a digital watermark generator for generating a digital 
markers, however, may assist the pirate in locating the watermark sequence responsive to a plurality of pre- 
digital watermark.) Nevertheless, the synchronization 40 determined watermark parameters and 
marker may comprise constant data that need not necessarily a quantizer of the perceptual coder responsive to the 
comprise the watermark such as data representing the copy- digital water mark generator and the watermark location 
right owner, the source, and the identity of the protected selector for inserting a digital watermark at selected 
work for example by title, track and artist, as some of these locations within an output bitstream, said output bit- 
should be well known from the protected work itself Title, 45 stream i ncm ding the perceptually coded information 
track and artist may be present for display, for example, in signal 

digital program delivery systems known in the art. 2 . Apparatus as recited in claim 1 wherein said transfor- 
In the design of apparatus for playing a watermarked mation function comprises a modified discrete cosine trans- 
original or pirate copy, a specific field (for example, the form for transforming from the time domain to the fre- 
encoding algorithm type) is inserted into the watermark. The 50 quency domain and outputting spectral coefficients for 
encryption key and the watermarking sequence are derived comparison with a perceptual threshold, a coefficient being 
from a private key issued to the purchaser at the point of below the threshold identifying that coefficient as a candi- 
distribution. A decoder apparatus associated with any player date location for a portion of a digital watermark, 
will receive the private key and decrypt the watermark to 3. Apparatus as recited in claim 1 wherein said transfor- 
extract the specific field. Play can cease shortly after play $s mation function comprises a discrete cosine transform for 
begins if the field cannot be found. transforming from the time domain to the frequency domain 
Now referring to FIG. 3, there is shown a decoder for and outputting spectral coefficients for comparison with a 
playing protected information according to the present perceptual threshold, a coefficient being below the threshold 
invention, the protected compressed information including identifying that coefficient as a candidate location for a 
hidden data such as a watermark as described above. A 60 portion of a digital watermark. 

compressed digital bitstream is received at point 300 and is 4. Apparatus as recited in claim 1 wherein said transfor- 

presented to a key extractor 305 and to a bitstream decoder mation function comprises an N point fast Fourier transform 

310. The key extractor 305 extracts a key used to encrypt the for transforming from the time domain to the frequency 

watermark. A key extractor is optional if the key is trans- domain and outputting spectral coefficients for comparison 

mitted in another way than in the compressed digital bit- 65 with a perceptual threshold, a coefficient being below the 

stream. The key extractor delivers the key, preferably in a threshold identifying that coefficient as a candidate location 

secure manner to a watermark detector 315. The bitstream for a portion of a digital watermark 
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5. Apparatus as recited in claim 1 wherein said logical 
gate of said watermark location selector comprises a logical 
AND gate, said logical AND gate being responsive to said 
binary sequence output of said random number generator 
and said watermarking sequence output of said watermark 
generator. 

6. Apparatus as recited in claim 1 wherein said digital 
watermark generator comprises an input comprising said 
plurality of predetermined watermarking parameters includ- 
ing play authorization data. 

I. Apparatus as recited in claim 6 wherein said plurality 
of watermarking parameters comprises one of data repre- 
senting an identity of a recipient of said protected informa- 
tion and data representing an identity of a perceptual coding 
transformation algorithm. 

8. Apparatus as recited in claim 6 wherein said plurality 
of watermarking parameters comprises one of data repre- 
senting an identity of a contractual arrangement and data 
representing an identity of the date of said contractual 
relationship. 

9. Apparatus as recited in claim 8 wherein said contractual 
arrangement comprises a lease and said plurality of water- 
marking parameters further comprises data indicative of an 
expiration date of said lease representing a date after which 
access to said protected information is to be denied. 

10. Apparatus as recited in claim 6 wherein said plurality 
of watermarking parameters is encrypted prior to insertion. 

II. Apparatus as recited in claim 1 wherein said digital 
watermark generator comprises means for encrypting a 
plurality of watermarking parameters with a key K related to 
said seed. 

12. Apparatus as recited in claim 1 wherein said digital 
watermark generator comprises means for encrypting a 
plurality of watermarking parameters with a key K unrelated 
to said seed. 

13. Apparatus as recited in claim 1 wherein said protected 
information comprises video data and said quantization has 
a step size selected according to the magnitude of errors that 
will not be visible. 

14. A method for inserting a digital watermark into 
protected information comprising the steps of 

perceptually encoding an information signal representa- 
tive of the protected information, 

selecting, responsive to the perceptual coding step, a 
location for inserting a digital watermark including the 
step of generating a binary sequence using a random 
number generator responsive to selecting a seed and a 
watermark overhead and providing said binary 
sequence to a logical gate, 

generating a digital watermark sequence responsive to a 
plurality of watermark parameters for input to said 
logical gate, and 

during perceptual coding, quantizing the information sig- 
nal to insert the generated digital watermark at selected 
locations within an output bitstream, said output bit- 
stream including the perceptually coded information 
signal. 

15. A method as recited in claim 14 wherein said percep- 
tual coding comprises a transformation function of a modi- 
fied discrete cosine transform for transforming from the time 
domain to the frequency domain and outputting spectral 
coefficients for comparison with a perceptual threshold, a 
coefficient being below the threshold identifying that coef- 
ficient as a candidate location for a portion of a digital 
watermark. 

16. A method as recited in claim 14 wherein said percep- 
tual coding comprises a transformation function of a discrete 
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cosine transform for transforming from the time domain to 
the frequency domain and outputting spectral coefficients for 
comparison with a perceptual threshold, a coefficient being 
below the threshold identifying that coefficient as a candi- 
s date location for a portion of a digital watermark. 

17. A method as recited in claim 14 wherein said percep- 
tual coding comprises a transformation function of an N 
point fast Fourier transform for transforming from the time 
domain to the frequency domain and outputting spectral 

10 coefficients for comparison with a perceptual threshold, a 
coefficient being below the threshold identifying that coef- 
ficient as a candidate location for a portion of a digital 
watermark. 

18. A method as recited in claim 14 wherein said step of 
15 selecting a watermark location comprises generating a ran- 
dom number binary sequence and logical ANDing an output 
of said random number generator and a candidate water- 
marking sequence. 

19. A method as recited in claim 18 wherein said step of 
20 generating a random number is responsive to inputting a 

seed and selecting a watermarking overhead. 

20. A method as recited in claim 14 wherein said step of 
generating a digital watermark comprises an input compris- 
ing said plurality of watermarking parameters including play 

25 authorization data. 

21. A method as recited in claim 20 wherein said plurality 
of watermarking parameters comprises one of data repre- 
senting an identity of a recipient of said protected informa- 
tion and data representing an identity of a perceptual coding 

30 transformation algorithm. 

22. A method as recited in claim 20 wherein said plurality 
of watermarking parameters comprises one of data repre- 
senting an identity of a contractual arrangement and data 
representing an identity of the date of said contractual 

35 relationship. 

23. A method as recited in claim 22 wherein said con- 
tractual arrangement comprises a lease and said plurality of 
watermarking parameters further comprises data indicative 
of an expiration date of said lease representing a date after 

40 which access to said protected information is to be denied. 

24. A method as recited in claim 20 wherein said plurality 
of watermarking parameters is encrypted prior to insertion. 

25. A method as recited in claim 19 wherein said digital 
watermark generating step comprises encrypting a plurality 

45 of watermarking parameters with a key K related to said 
seed. 

26. A method as recited in claim 19 wherein said step of 
generating a digital watermark comprises encrypting a plu- 
rality of watermarking parameters with a key K unrelated to 

50 said seed. 

27. Amethod as recited in claim 14 wherein said protected 
information comprises video data and said quantization step 
has a step size selected according to the magnitude of errors 
that will not be visible. 

55 28. Apparatus for inserting a hidden data stream into a 
perceptually encoded media stream comprising 

an N point transform for outputting N coefficients and 
a quantizer for selecting a scale factor, quantizing the 
input N coefficients and inserting the hidden data 
60 stream in indexed locations of the perceptually encoded 
media stream responsive to said N point transform, said 
indexed locations being determined responsive to 
selection of a seed and a watermark overhead for input 
to a random number generator for generating a binary 
65 sequence. 

29. Decoder apparatus for detecting a watermark for 
authorizing play, said watermark being generated responsive 
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to an input comprising a plurality of watermarking param- 
eters including play authorization data and being located 
responsive to selection of a key and a watermark overhead 
comprising 

a key receiver for receiving a key relating to said location 5 
of a watermark, 

a watermark detector responsive to the key receiver for 
locating the watermark and for analyzing if the water- 
mark contains play authorization data and 

10 

a presentation circuit for presenting a decoded informa- 
tion signal responsive to the watermark detector when 
said play authorization data of said watermark autho- 
rizes play. 

30. Decoder apparatus for detecting a digital watermark in 
an information signal, the digital watermark being generated 
responsive to an input comprising a plurality of watermark- 
ing parameters including play authorization data and being 
located responsive to selection of a key and a watermark 
overhead and including the name of a customer and the date 2Q 
and location of a transaction, and for outputting the water- 
mark for tracing an unlicensed copy of the information 
signal to a pirate, the unlicensed copy including said water- 
mark. 

31. A method for inserting a digital watermark into 25 
protected information comprising the steps of 

perceptually encoding an information signal representa- 
tive of the protected information, 

selecting, responsive to the perceptual coding step, a 
location for inserting a digital watermark, 30 

generating a digital watermark including a plurality of 
watermark parameters, and 

during perceptual coding, manipulating perceptual coding 
coefficients to insert the generated digital watermark at 
selected locations within an output bitstream, the out- 35 
put bitstream including the perceptually coded in for- 
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mation signal, said selection of locations for the gen- 
erated watermark being responsive to a comparison of 
said perceptual coding coefficients with a calculated 
perceptual threshold and includes the steps of selecting 
a watermark overhead and a seed for input to a random 
number generator. 

32. A method for inserting a digital watermark into 
protected information as recited in claim 31 wherein said 
protected information comprises audio information. 

33. A method for inserting a digital watermark into 
protected information characterized by the steps of 

determining a candidate digital watermark sequence, 
selecting a digital watermark overhead for controlling 
generation of a binary output sequence for logical 
combination with said candidate digital watermark 
sequence and 

locating, responsive to said logical combination and said 
selected digital watermark overheads, a digital water- 
marking sequence generated responsive to a plurality of 
watermark parameters into said protected information 
data signal. 

34. A method as recited in claim 33 wherein said step of 
selecting a digital watermark overhead provides an input to 
a random number generator whose output is provided to a 
logical gate for logical combination with said candidate 
digital watermark sequence. 

35. A method as recited in claim 34 wherein said output 
of said random number generator is further determined by 
the selection of a seed as a further input. 

36. A method as recited in claim 34 wherein said digital 
watermark overhead is selected as one of a percent or a 
probability of a binary output of the random number gen- 
erator being equal to one. 

***** 
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